K T K TEO et al: AGENT-BASED OPTIMIZATION FOR MULTIPLE SIGNALIZED INTERSECTION

Relieving urban traffic congestion has always been an urgent call in a dynamic traffic network. The objective of this research is to control the traffic flow within a traffic network consists of multiple signalized intersections with traffic ramp. The massive traffic network problem is dealt through Q-learning actuated traffic signalization (QLTS), where the traffic phases will be monitored as immediate actions can be taken during congestion to minimize the number of vehicles in queue. QLTS is tested under two cases and has better performance than common fixed-time traffic signalization (FTS). When dealing with the ramp flow, QLTS has flexibility to change the traffic signals according to the traffic conditions and necessity. Keywords-Disturbance, Multi-agent, Q-learning, Traffic Signalization, Traffic Flow Optimization. I. INTRODUCTION Urban area is a large population coverage area with complex traffic network. The traffic network plays a major role in civilization development, as it serves as the communication or travel linkage between places. Relieving traffic predicament from the deteriorating congestion has always been an urgent call to the urban society. One of the major traffic network dynamicity comes from the progressive inflow from any possible branches, recognizable as a traffic flow disturbance which might hit the traffic bottleneck and cause exceeding road capacity. Failure in coping with the sudden vehicles increment usually ends up in traffic immobility. The congestion effect at a single traffic will be carried forward and affecting the nearby intersections. It is therefore important to look into the congestion in a network rather than a single intersection problem. Flexible traffic signals are required at multiple intersections to react upon the traffic needs when the congestion strikes [1]. The cost of traffic congestion is not limited to the time lost during the travelling, but the possible consequences and the chain effect caused by the congestions. The basic traffic modeling provides an insight for traffic estimation. On a contrary, precise and robust dynamic traffic modeling is costly and restricted to particular geographical areas and varying parameters. This work suggested agentbased traffic management to control the traffic signals at double intersections. Suiting the frequent changing traffic environment, the agent plays an important role in autonomous data collection and dynamics studies of traffic intersections for future self-judgment of green time duration. The situation whereby disturbance occurs at the intersection is included in this work to depict the network dynamicity and to test the robustness of the developed system towards the changing network conditions. The agent is able to change the traffic signals according to the traffic needs. II. OVERVIEW OF TRAFFIC CONTROL SYSTEMS Under the condition of limited land and the difficulty to rebuild the road infrastructure, one of the best solutions to the current traffic scenario is to study and design a heuristic traffic flow controller to enhance traffic smoothness in the congested urban area. Several important parameters in the traffic flow control study are extracted based on the official reports and manuals from Malaysian Public Works Department (JKR) [2], as well as Transportation Research Board’s Highway Capacity Manual, as defined below [3]: • Vehicles in queue: Total number of vehicles that line up in front of the intersection waiting to be given the permission for passing the intersection. • Traffic phase: Group of traffic lanes given the same traffic signal to avoid conflicts with other traffic lanes. • Cycle time: Time for the traffic light signals to be circulated once after all the phases have their turn of green signals. • Lost time: Time lost between the interchange of the signals or the time lost due to the driver’s behavior at the intersection, which may increase the number of interchanges between the signals. The most common traffic management system is the fixed-time traffic signalization (FTS) where duration of each traffic signal is fixed. Due to the increasing vehicles on road, the traffic signalization is improved with predetermined strategic management according to the collected traffic data [4]. The Webster’s traffic model [4] has a large influence in the designs of traffic signalization system, as many researchers and department of transportation are still using the model of Webster as the basis of their practical design procedure. JKR published its guidelines in traffic signal designs using the implementation of Webster’s study [2]. However, predetermined FTS does not have the ability to K T K TEO et al: AGENT-BASED OPTIMIZATION FOR MULTIPLE SIGNALIZED INTERSECTION DOI 10.5013/IJSSST.a.15.06.10 91 ISSN: 1473-804x online, 1473-8031 print react towards the dynamic environment such as unforeseen disturbances and delays in the traffic network. Therefore, heuristic control and adaptation to the dynamic changes of the traffic flow is important to help prevent congestion. Various artificial intelligence (AI) methods have been attempted in the traffic system due to limiting performance of the existing traffic controllers. In recent years, the common algorithms implemented include fuzzy logic (FL), genetic algorithm (GA) and reinforcement learning (RL). FL based traffic control and optimization had been researched to slow down the accumulation rate of waiting vehicles at the intersections [5-7]. Azimirad et al. (2010) worked out the queue length and the vehicles’ waiting time at the intersection as the input parameters of traffic signal timing to optimize the traffic flow using FL embedded control system [5]. However, the simulation was performed at an isolated intersection. The effects of other intersections towards the simulated traffic intersection are not being considered. The FL based traffic management strategy requires precise definition of the membership functions for mapping the input and the output, which causes the shortcoming of not being able to act when the data input is not within the range. GA based traffic optimizer has been proposed specifically for the purpose of optimizing the oversaturated traffic conditions [8-10]. Sánchez et al. (2008) developed microscopic traffic model using cellular automata (CA) where it is optimized by a developed GA [8]. CA served as the fitness evaluation functions, with the chromosomes of GA as the input. GA can learn and evolve from the environment. However, its chromosome representation will become more complicated when the traffic problem is expanded. The number of chromosomes within GA is highly depending on the problem size. RL is introduced for its ability to learn from the traffic problem environment and build up memory for future decision and traffic optimization [11-13]. Arel et al. (2010) developed a RL based traffic optimization control, implemented as a multi-agent system in the traffic environment [11]. The traffic flow is the input and state of the Q-learning (QL) agent in every intersection. QL is one of the variance in RL. Liu and Ma (2007) suggested Dyna-Q algorithm but is not implemented in multi-agent system for the optimization purpose [12]. This work is interested in exploring the potential of QL in a traffic network of nonsingular intersections including the situations whereby traffic disturbance exists. The development of Q-learning actuated traffic signalization (QLTS) algorithm will be discussed in the next section. III. Q-LEARNING ACTUATED TRAFFIC SIGNALIZATION Intelligent traffic management is able to extend the existing FTS to cope with the overwhelming dynamic traffic flows in the urban traffic network. QL values the effects of actions taken and the resulting state in a traffic environment, which is very much necessary for a learning agent to react appropriately in a fast changing traffic environment [14-15]. QL tends to seek for the best solutions through action that returns the highest reward. For the process of managing the traffic signal, QL has to be defined precisely, in order for the algorithm to be able to interact with the traffic environment and decide the best actions that can counteract the severe traffic congestions. The state-action pairs of the QL are the main part that determines the robustness of the algorithm. Besides the state-action pairs, the reward function is also a crucial function that will decide the policies of the algorithm in its learning process. In this study, the level of vehicles in queue at each intersection is chosen to be the state of the designed QL. The vehicles in queue are produced from the difference between the traffic inflow and the traffic outflow. Equation (1) is the calculation for the vehicles in queue i Q , where i is the current intersection and j is the source of the departure traffic. i j z , is the traffic flow rate from intersection j into i while i y is the average outflow at i . In (2), C is the cycle length in the developed model, ∑ z is the total of traffic flow rate into the intersection i , and lastly i g is the effective green signal duration. ) ( ) ( , i i i j i g y C z Q − = (1)